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1  Introduction 


New  computer  and  communications  technologies,  coupled  with  advances  in  microscope  and  manipulator 
design,  have  made  it  possible  to  extend  the  capabilities  of  neurosurgeons  into  previously  inaccessible 
locations,  such  as  under-served  rural  and  inner  city  environments  or  battlefield  situations.  We  are 
constructing  the  Integrated  Remote  Neurosurgical  System  (IRNS),  a  remotely-operated  neurosurgical 
microscope  with  high-speed  communications  and  a  surgeon-accessible  user  interface.  The  ERNS  will 
allow  high  quality  bidirectional  mentoring  in  the  neurosurgical  suite.  Our  research  goals  are  twofold: 
First,  to  provide  an  integrated  training  environment,  an.d  second,  to  develop  a  clinical  system  allowing 
a  remote  neurosurgeon  to  lend  expertise  to  the  OR-based  neurosurgical  team. 

The  IRNS  uses  a  generic  microscope/ transport  model,  allowing  a  wide  range  of  operating  room  equipment 
to  be  controlled.  In  our  case,  the  equipment  is  the  Carl  Zeiss  [1]  MKM  surgical  microscope/transport. 
Translation  firom  generic  commands  to  hardware-specific  transport  commands  is  performed  by  SuMIT 
(the  Surgical  Manipulator  Interface  Translator).  A  SuMIT  interface  has  been  developed  for  the  MKM. 

To  the  remote  surgeon,  the  most  important  aspect  of  the  IRNS  is  the  Remote  Planning  and  Navigation 
Workstation.  The  workstation  incorporates  surgical  planning  capabilities  and  can  view  real-time  video 
from  both  the  microscope  and  an  overhead  video  camera.  It  also  incorporates  the  ability  to  remotely 
position  the  microscope  head  and  to  its  focus  depth.  The  remote  workstation  includes  a  3-D  input 
device  which  uses  tracking  technology  to  allow  the  remote  surgeon  to  intuitively  position  the  microscope. 
Bidirectional  audio  and  image  archiving  are  also  implemented. 

The  training  goals  of  our  research  are  also  embodied  in  the  remote  workstation.  Real-time  simulation 
of  the  microscope  transport  is  provided  by  the  commercially  available  Telegrip  simulation  package  from 
Deneb  Robotics  [2].  Telegrip  is  used  to  produce  a  realistic  view  of  what  the  remote  surgeon  would  see 
on  the  planning  workstation.  The  simulation  permits  pre-surgical  simulation,  post-surgical  critique,  and 
training  for  surgeons  without  access  to  an  actual  microscope  transport  system. 

The  components  of  the  IRNS  are  integrated  using  ATM  [3]  switching  to  provide  low  latency  data 
transfer.  Guidelines  have  been  devised  to  ensmre  safe  system  operation  both  during  normal  operation 
(i.e.  transport  control  handoff,  collision  avoidance)  and  under  error  conditions  (i.e.  loss  of  communi¬ 
cation,  transport  failure,  OR  emergencies).  A  registration  system  has  been  developed  to  ensure  precise 
monitoring  of  the  patient  and  microscope. 

The  IRNS  provides  an  opportunity  to  assess  the  benefits  of  remote  surgical  mentoring.  By  integrat- 
ing  high  performance  workstations  and  microscope  csirriers,  the  system  demonstrates  the  usefulness  of 
communications  emd  computing  resources  in  improving  access  to  specialized  surgical  facilities.  This  re¬ 
search,  along  with  the  more  sophisticated  systems  that  will  follow,  will  serve  as  a  foundation  and  testing 
platform  for  extending  the  surgeon’s  skills  without  regard  to  time  zone  or  geographic  boundaries. 

This  document  will  describe  the  IRNS  in  its  current  state,  including  the  system  architecture,  commu¬ 
nications  architecture,  safety  guidelines,  user  interface,  and  hardware.  We  will  then  discuss  the  future 
research  and  development  goals  for  the  IRNS. 


2  IRNS  System  Architecture 

2.1  Overview 


The  IRNS  consists  of  two  worksites,  the  operating  room  and  the  remote  workstation.  They  are  con¬ 
nected  via  an  ATM  network  (see  figure  1).  Each  worksite  hosts  several  architectural  elements,  such 
as  workstations  and  cameras.  Locally,  these  elements  are  interconnected  by  ATM,  Ethernet,  or  RS-232 
connections,  and  are  ultimately  linked  by  a  site  workstation.  The  ATM  link  is  responsible  for  carrying 
aU  video,  audio,  and  control  data.  The  system  is  currently  simulating  the  MKM  and  optical  registration 
system  in  Telegrip  until  the  hardware  is  acquired.  The  models  will  still  be  retained  for  the  simulation 
and  training  goals  of  the  project.  The  ATM  network,  room  camera  software,  audio  communications, 
cind  user  interface  are  all  present. 


2.2  Communications  Design 

.Asynchronous  Transfer  Mode  (ATM)  networking  is  an  emerging  communications  technology  in  which 
fixed-length  ’’cells”  of  data  are  transferred  at  speeds  of  over  2.3  Gbits/sec  [4].  The  primary  advantage  of 
-ATM  is  that  it  was  specifically  designed  for  simultaneous  data,  voice  and  video  transmission  -  precisely 
the  characteristics  required  for  telemedicine. 

We  have  implemented  a  four-node  ATM  switch  network  that  incorporates  both  WAN  and  LAN  elements. 
EEigh  performance  workstations  are  linked  together  using  100  Mbit /sec  LATM  connections  through  a 
Newbridge  Networks’  Model  36150  8x8  ATM  switch  for  file  transfer  and  distributed  computing  tasks. 
-Another  LATM  connection  links  our  lab  with  the  principal  Operating  Rooms  in  the  University  of  Vir¬ 
ginia  HecJth  Science  Center.  This  connection  is  used  to  transmit  image  data,  JPEG  video  and  audio 
information  between  the  lab  and  the  0/R. 

For  transmission  of  image  data  and  control/feedback  information,  both  locally  and  remotely,  we  use 
TelRIP  [5],  a  general  purpose  data  exchange  system.  TelRIP  uses  a  data-centered  approach  to  modu¬ 
larity,  meaning  that  programs  communicate  with  one  another  by  specifying  the  fypes  of  data  they  wish 
to  send  or  receive.  An  advantage  is  that  our  processes  can  continue  to  run  without  knowing  the  state 
of  other  processes  on  the  network.  This  approach  also  allows  us  to  standardize  the  interfaces  to  devices 
and  controls  without  predetermining  the  exact  implementation  of  each  interface. 


2.3  O/R  Worksite 

The  O/R  worksite  consists  of  the  following  components: 

•  -A  surgical  stereomicroscope  mounted  on  a  robotic  transport, 

•  One  or  more  ‘bird’s  eye’  view  cameras  on  pan/tilt  heads, 

•  -An  optical  registration  system,  and 

•  -A  workstation  to  display  and  annotate  pre-surgical  eind  inter-operative  medical  imagery. 

Live  video  from  both  the  stereomicroscope  and  the  O/R  view  camera(s)  is  routed  through  JPEG  com¬ 
pression  hardware  before  transmission  across  the  ATM  link  and  is  subsequently  decompressed,  again 
in  hardware,  at  the  remote  location.  The  O/R  ’’bird’s  eye”  view  cameras  were  included  to  give  the 
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Figure  1  :  System  Overview 


remote  stirgeon  feedback  on  the  situation  in  the  0/R.  This  type  of  feedback  is  known  to  improve  the 
interactivity  of  collaborative  systems  [6]  by  giving  the  remote  user  a  sense  of  presence  at  the  local  site 
(in  this  case,  the  0/R). 

The  optical  registration  system,  a  simulation  of  Image  Guided  Technologies^  Flashpoint  5000  [7],  is  also 
interfaced  to  the  overall  system  via  the  central  workstation.  This  component  is  used  to  calculate  the 
relative  positions  of  the  robot  base  and  patient  by  sensing  targets  which  axe  placed  on  them.  Locations 
of  these  objects  are  represented  internally  as  a  series  of  coordinate  systems.  Using  these  locations,  as 
well  as  other  information  gathered  by  the  system,  we  can  monitor  the  location  of  the  patient  relative  to 
the  microscope. 

The  workstation  can  display  views  from  the  stereomicroscope,  pre-surgical  MR  and  CT  images,  and  3D 
surgical  planning  images.  Users  in  the  0/R  can  interactively  slice  through  patient  data,  zoom  in  on 
images,  and  view  a  rendered  volume  of  the  brain.  In  addition,  verbal  communication  between  the  0/R 
and  the  remote  site  is  provided  through  microphones  and  ATM-linked  audio. 


2.4  Remote  Worksite 

The  remote  worksite  consists  of  a  graphics  workstation  providing: 

•  Microscope  control, 

•  Robotic  transport  control, 

•  Video  from  the  microscope  and  0/R  view  cameras,  and 

•  Surgical  planning  facilities. 

Microscope  controls  which  are  extended  to  the  remote  workstation  include  manual  focus  and  zoom 
adjustments  as  weU  as  the  automatic  focusing  control.  These  controls  can  be  activated  by  using  verbal 
commands  to  the  system's  speech  recognition  software,  Speech  Systems  Phonetic  Engine  500  [8]. 

Microscope  transport  positioning  control  can  be  performed  in  three  ways: 


•  Graphically  -  The  surgeon  can  manipulate  the  graphical  user  interface  (GUI)  to  translate  the 
microscope. 

•  Verbally  -  The  surgeon  can  issue  commands  to  the  microscope  by  speaking. 

•  Analogically  -  The  surgeon  can  use  magnetically  trcLcked  spatial  input  devices  to  effect  the  same 
orientation  on  the  microscope. 


Facilities  on  the  layout  of  the  GUT  allow  the  surgeon  to  pan  the  microscope  and  zoom  using  arrow 
buttons  and  a  thumbwheel-style  widget.  These  buttons  can  be  activated  either  by  using  the  mouse  or  by 
pressing  the  touchscreen.  In  addition,  speech  recognition  software  allows  the  remote  surgeon  to  control 
panning  and  zooming  of  the  microscope  simply  by  speaking  ”pan  left”,  "zoom  in”,  etc. 

The  current  version  of  the  system  uses  "props”  [9],  which  are  real-world  objects  with  magnetic  trackers 
embedded  within  them.  There  are  two  props:  one  which  represents  the  microscope  (a  stylus)  and  one 
w’hich  represents  the  patient's  head  (a  brain  model).  By  monitoring  the  orientation  of  these  trackers,  the 
system  can  give  the  actual  microscope  the  same  orientation  with  respect  to  the  patient.  This  interface 


technique  has  been  in  use  at  our  lab  for  some  time  (on  a  prototype  surgical  planning  tool),  and  has 
received  enthusiastic  support  from  both  surgeons  and  residents  for  its  ease  of  use. 

Feedback  from  the  OR  is  provided  in  video  and  audio  forms.  The  remote  surgeon  is  presented  with  live 
(30fps)  video  from  the  microscope  and  the  0/R  camera  views.  A  graphical  simulation  of  the  microscope 
transport’s  current  position  is  also  shown,  to  give  the  surgeon  a  better  feel  for  the  spatial  configuration 
of  the  patient,  microscope,  and  transport.  Snapshots  of  the  view  through  the  microscope  can  be  taken 
and  saved  along  with  the  positioning  information  necessary  to  return  the  transport  to  the  snapshot  view. 
To  return  the  microscope  to  the  position  from  which  snapshots  were  taken,  commands  may  be  issued 
via  the  graphical  user  interface  or  by  verbal  command. 

The  surgical  planning  and  audio  facilities  are  identical  to  those  on  the  0/R  workstation. 


3  Safety  Precautions 


The  responsibility  of  overall  system  safety  is  divided  between  four  different  methods: 

•  Restriction  of  Robot  Motion  -  The  robot’s  motion  is  limited  to  a  specified  safety  envelope. 

•  Registration  of  Critical  Objects  -  The  precise  locations  of  the  patient  and  microscope  must  be 
known  at  all  times. 

•  Implementation  of  an  Operation  Protocol  -  Users  must  follow  a  rigid  set  of  procedures  during  the 
system’s  use, 

•  Installation  of  Safety  Hardware  -  A  set  of  safety  processors  ensure  that  the  robot  is  functioning 
normally. 

To  avoid  collisions,  the  robot’s  motion  is  restricted  first  to  a  working  volume,  the  set  of  all  points  which 
can  be  physically  reached  by  the  robot.  The  Telegrip  package  can  graphically  display  this  work  volume. 
On  a  second,  smaller  level,  the  working  volume  is  constrained  by  the  patient  and  the  operating  area. 
Generally  speaking,  motion  will  be  constrained  within  a  spherical  safety  envelope  above  the  operating 
area.  Presently,  more  detailed  specifications  on  the  MKM  axe  being  acquired  to  accurately  calculate 
this  envelope.  Registration  becomes  necessary  in  order  to  calculate  the  safety  envelope  and  monitor  the 
locations  of  the  patient  and  robot.  This  registration  system  has  been  implemented  in  our  simulation. 

The  registration  system  consists  of  two  components:  the  optical  registration  system  in  the  0/R  and  the 
user  interface  at  the  remote  site.  While  the  optical  system  measures  both  the  locations  of  the  patient 
and  robot  base,  the  user  interface  provides  the  surgeon’s  intended  location  of  the  microscope  relative 
to  the  patient.  These  coordinate  systems  are  then  combined  to  provide  a  measurement  of  the  location 
of  the  robot  with  respect  to  the  patient.  Motion  commands  from  the  user  interface  will  be  checked  at 
the  0/R  for  safety  limitations  based  on  this  registration  to  ensure  that  the  robot  does  not  move  outside 
of  its  safety  envelope.  SuMIT  then  performs  the  low-level  translations  necessary  to  convert  positions  in 
these  internal  coordinate  systems  into  robot-specific  motions. 

Another  important  problem  in  maintaining  the  safety  of  the  robot’s  motion  is  the  avoidance  of  singular¬ 
ities,  These  are  configurations  of  the  robot  arm  where  certain  directions  of  motion  may  be  unattainable. 
They  often  correspond  to  points  on  the  boundary  of  the  robot’s  workspace,  out  of  its  maximum  reach. 

The  key  to  avoiding  singularities  is  to  keep  the  working  volume  of  the  robot  within  its  dextrous  workspace, 
w'hich  is  the  set  of  ail  points  that  the  robot  can  reach  with  an  arbitrary  orientation  of  the  tool.  Singu¬ 
larities  are  determined  by  the  structure  and  parameters  of  a  particular  robot.  They  can  be  calculated 


and  shown  on  a  package  such  as  Telegrip.  The  IRNS  software  is  designed  to  recognize  singularities 
and  provide  feedback  to  the  surgeon  should  the  robot  approach  one.  This  feedback  is  currently  given 
by  color-coding  the  affected  joints  in  Telegrip,  but  will  be  provided  later  through  force  feedback  in  the 
remote  surgeon’s  input  device. 

Both  the  0/R  and  remote  sites  can  control  the  robot,  although  not  at  the  same  time.  Handoff  of  control 
between  the  two  sites  will  be  negotiated  between  the  remote  surgeon  and  the  team  in  the  0/R  site. 
A  single  ’’dead  man’s  switch”  interface  ensures  that  only  one  party  has  control  of  the  microscope  at 
any  given  time.  The  robot  is  in  a  stopped  state  upon  startup  and  shutdown  of  the  system.  Should 
communication  between  the  remote  and  local  parties  be  severed,  aborted  commands  are  retried  until 
some  timeout.  The  local  team  in  the  0/R  always  has  the  option  to  switch  to  controlling  the  robot 
themselves.  In  short,  the  local  team  is  able  to  continue  the  procedure  should  communication  with  the 
remote  surgeon  be  lost. 


The  MKM  system  is  also  equipped  with  a  number  of  safety  processors  which  monitor  the  robotic  arm 
for  failures  such  as  excessive  velocity  and  range  limits.  Procedures  for  recovery  from  failures  related  to 
these  safety  processors  will  be  followed  in  accordance  with  the  MKM  manuals  [10]. 


4  Graphical  User  Interface  Design 

4.1  Task  Analysis 


Feedback  from  surgeons  and  residents  has  been  sought  throughout  the  project’s  development.  Observa¬ 
tions  of  surgeons  using  robotic  microscopes  have  been  conducted  at  the  University  of  Virginia  hospital, 
and  have  also  been  reviewed  on  video  logs.  These  observations  are  critical  in  determining  the  current 
deficiencies  in  the  usability  of  the  surgical  microscope  as  well  as  the  interaction  style  to  which  the  sur¬ 
geons  have  become  accustomed.  Our  system  can  correct  some  of  the  observed  problems  (operating  table 
movement,  jerky  motion  of  the  microscope)  because  of  the  increased  fiexibility  afforded  by  its  remote 
interface  and  registration  system.  Interviews  were  also  conducted  to  determine  what  major  functions 
are  desired  by  surgeons.  Features  which  came  out  of  this  process  include  speech  recognition  capabilities, 
a  simulated  view  of  the  microscope  (through  Telegrip),  and  the  snapshot  facility. 

In  addition  to  the  design  criteria  specified  by  the  task,  current  research  concepts  from  the  areas  of  robotic 
control  paradigms  and  human-computer  interaction  were  included  in  the  design.  The  selection  of  input 
devices  and  the  graphical  user  interface  layout  and  interaction  style  were  influenced  in  part  by  previous 
research  [11]  [12]. 


4.2  Interface  Design 

A  layout  for  the  remote  console  has  been  designed  to  conform  to  the  needs  of  the  simgeon.  It  represents 
the  result  of  feedback  received  from  surgeons  and  lay  users  as  well  as  from  our  ongoing  task  analysis. 
The  GUT  currently  includes: 

•  A  large  window  which  provides  the  view  from  the  microscope.  Built  on  top  of  this  window  is  a 
’’virtual  trackball”  [13]  which  allows  the  surgeon  to  rotate,  pan,  and  zoom  the  virtual  microscope 
relative  to  any  point  in  space.  This  controller  serves  as  a  backup  to  the  tracked  3-D  input  devices. 

•  A  second,  smaller  window  provides  live  video  from  a  bird’s  eye  view  of  the  operating  room.  Controls 
on  the  window  allow  the  surgeon  to  pan  the  camera  in  4  directions  as  well  as  zoom. 


Figure  2:  Layout  of  remote  workstation  interface 


•  Two  more  windows  provide  views  of  the  patient’s  MRI  data  and  a  rendered  image  of  a  segmented 
volume  of  this  data.  The  surgeon  can  easily  zoom  and  slice  through  the  data  using  thumbwheel- 
style  widgets  beneath  the  windows. 

•  A  ’’snapshot”  facility  allows  the  surgeon  to  record  still  images  from  the  microscope  camera  and 
store  them.  A  separate  window  normally  displays  the  most  recent  snapshot,  but  a  history  menu  of 
thumbnail  images  lets  the  user  select  an  older  snapshot  to  display.  Upon  command,  the  microscope 
transport  can  return  to  the  position  from  which  any  snapshot  was  taken. 


A  speech  recognition  system  has  been  integrated  into  this  prototype  for  use  in  issuing  verbal  commands. 
Most  of  the  GUI  functions  now  have  verbal  equivalents,  such  as  panning  the  microscope  and  taking 
snapshots  of  the  current  microscope  view.  Response  has  been  positive  from  our  primary  consulting 
surgeon  and  casual  visitors  to  our  lab,  so  this  means  of  interaction  is  continually  being  enhanced  to 
include  more  commands. 

Currently,  the  user  interface  is  connected  to  the  Telegrip  robot  simulation  package  through  the  use  of 
the  TelRIP  networking  software  mentioned  above.  The  user  is  able  to  remotely  control  the  simulated 
robot  through  the  GUI.  See  figure  2. 


4.3  Robot  Simulation 


A  model  of  the  robotic  microscope  transport  was  developed  in  Telegrip  so  that  the  actions  of  the  robot 
could  be  simulated  when  it  was  not  actually  available.  It  is  a  complete  simulation  with  closed-form 
inverse  kinematics  linked  into  Telegrip  through  a  shared  library.  The  closed-form  solution  is  more  rapid 
than  the  iterative  one  and,  more  importantly,  is  capable  of  determining  the  singular  positions  within  the 
workspace  of  the  microscope  transport.  This  model  can  be  manipulated  locally  through  the  Telegrip  UI 
and  remotely  through  any  TCP /DP  link.  Various  other  components  of  the  0/R  have  also  be  modeled  in 
this  simulation  including  the  camera,  optical  tracker  and  patient. 


5  Ongoing  Work 


Even  though  the  props-based  interface  currently  connected  to  the  remote  workstation  is  intuitive  and 
easy  to  use,  it  contains  a  great  deal  of  instability  due  to  the  tracking  technology  and  is  incapable  of 
providing  active  force  feedback.  Therefore,  we  intend  to  replsice  it  with  a  force-reflecting  hand  controller. 
This  type  of  controller  will  give  the  surgeon  an  intuitive  way  to  guide  the  microscope  transport,  simply 
by  pushing  or  pulling  on  a  handle  mounted  on  a  table-top  robot.  Force  reflection  will  be  used  to  indicate 
boundaries  outside  of  which  the  remote  surgeon  may  not  move  the  microscope.  The  hand  controller  could 
then  be  combined  with  another  tool  to  allow  the  two-handed  interaction  that  the  current  props-based 
interface  now  provides. 

The  system  will  soon  have  the  capability  of  whiteboarding.  This  will  allow  users  at  either  the  local  or 
remote  sites  to  ’’sketch”  on  images  using  a  touchscreen  or  mouse.  These  annotations  will  be  reproduced 
simultaneously  on  the  other  side  of  the  ATM  link,  allowing  the  remote  surgeon  and  the  local  0/R  team 
to  communicate  visually  as  well  as  verbally.  Whiteboarding  software  has  been  developed  and  will  soon 
be  integrated  into  the  remote  and  local  user  interfaces. 

In  addition,  the  trackball-style  interface  used  in  the  GUI  of  the  remote  console  will  probably  be  replaced 
with  additional  whiteboarding  capabilities.  This  ability  to  overlay  graphics  onto  our  video  streams  would 
open  the  way  for  more  complex  enhancements  such  as  overlaying  of  segmented  images  and  augmented 
reality  elements  [14]. 

Because  the  IRNS  is  still  under  development,  we  have  not  conducted  any  formal  experiments.  When 
we  have  a  complete  prototype  implementation  (expected  by  December  1996),  we  plan  to  conduct  trials 
with  our  consulting  physician  in  order  to  take  measurements  on  network  loading,  operational  efficacy, 
learning  time,  and  positional  accuracy.  We  have  a  working  simulation  with  some  real  components,  and 
need  only  the  MKM  and  optical  registration  system  to  move  from  simulation  to  prototype. 
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